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Method of determining a structure of a moving object 



5 The present invention relates to the field of digital imaging. In particular, 

the present invention relates to a method of determining a structure of a moving object 
from an at least two dimensional data set, an image processing device and a computer 
program for an image processing device. 

10 The anatomy of the human coronary tree is widely consistent between 

normal subjects. It always consists of three main branches called left anterior 
descending (LAD), left circumflex (LCX) and right coronary artery (RCA). The main 
variation in topology to be found is a dominance of either left or right branch supplying 
the apex of the heart, as described in M.D. Cerqueira et al. "Standardized Myocardial 

15 Segmentation and Nomenclature for Tomographic Imaging of the Heart" Circultation 
105: 539-542, 2002 which is hereby incorporated by reference, attaching more sub- 
branches to either side. Geometric properties of segments, however, may vary 
significantly between subjects, for example, due to differences in size and shape of the 
ventricles of the heart. 

20 Document C. Chalopin, I.E. Magnin, G. Finet "Automatic labeling of the 

coronary tree using a three-dimensional reference prior Model" Computers in 
Cardiology 25, p. 761-764.1998 discloses the generation of two dimensional reference 
models of either left or right coronary tree from a given three-dimensional model, each 
of which having a different projection angle. The best fitting reference model is used to 

25 identify a coronary main branch in geographic projections and to label the segments of 
the image. In other words, the best fitting reference model is used to identify the 
respective branches of the human coronary tree in an image. 

It is an object of the present invention to provide for an improved 
30 prediction of properties of a moving object such as, for example, the human heart. 

The above object may be solved by a method of determining a structure 
of a moving object from an at least two-dimensional data set according to claim 1. 
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According to this method, a model of the structure relating to the properties of interest 
is applied to the data set. Then, an adaptation of the model to the data set is performed. 
Then, on the basis of the adapted model, a location of at least one portion of the 
structure of the moving object is estimated. 
5 In other words, with respect to the example of the coronary tree of a 

human heart, a model of the coronary tree, which is, for example, a statistic deformable 
model based on preceding measurements, is applied to, for example, a two-dimensional 
image, such as an x-ray angiography image. Then, this model, which may be a three- 
dimensional model, is adapted to the two-dimensional image representing the data set. 

10 This adaptation may, for example, be performed by a parameter variation. Then, a 

location of at least one portion of the structure, for example of a portion of the coronary 
artery with a total occlusion, which renders the vessel invisible, since no contrast agent 
is supplied to this region, is predicted by using the adapted model. 

Advantageously, this exemplary embodiment of the present invention 

15 may allow for improved diagnostics of, for example, occlusions in the artery tree or an 
improved detection of side branches of the coronary tree, since, on the basis of the 
model, a location of at least one portion of the coronary tree or the artery tree may be 
predicted. 

According to another exemplary embodiment of the present invention as 
20 set forth in claim 2, the adapted model is overlaid onto the image of the coronary tree 
such that an image may be displayed showing the actual measured image and the 
overlaid model, such that it can easily be determined where a vessel such as a lumen 
should be and whether it is not visible due to, for example, an occlusion, or where, for 
example, a side branch of the lumen may be expected. This may allow, for example, for 
25 improved catheter intervention. 

According to another exemplary embodiment of the present invention as 
set forth in claim 3, parameters of the model are adapted on the basis of a similarity of 
the model to the structure, which may allow for a simple and efficient adaptation of the 
model. 

30 According to another exemplary embodiment of the present invention as 

set forth in claim 4, an image quality of an image displayed may be improved by 
superimposing images of the object of interest taken at different points in time or with 
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different projections. For superimposing images taken at different points in time or 
projections, the adapted model may be used as a reference, allowing for a registration of 
these images. 

According to another exemplary embodiment of the present invention as 
5 set forth in claim 5, the model is a deformable model and the adaptation of the model is 
performed by an energy minimization of an internal and an external energy of the 
model, which allows for a fast adaptation of the model. 

According to another exemplary embodiment of the present invention as 
set forth in claim 6, the model is a statistical model of a coronary tree of a human heart 
10 and the data set relates to x-ray angiography data. 

According to another exemplary embodiment of the present invention as 
set forth in claim 7, an image processing device is provided, allowing to estimate a 
location of at least one portion of a structure of an object of interest, which, for 
example, is not discernible in an image representing a two-dimensional data set. This 
15 estimation is performed on the basis of a model of the structure of interest adapted to 
the image, i.e. the data set. 

Further exemplary embodiments of the image processing device are 
provided in claims 8 and 9. 

According to another exemplary embodiment of the present invention as 
20 set forth in claim 9, a computer program is provided allowing for an estimation of a 
location of at least one portion of a structure of a moving object of interest by using a 
model of the structure of interest adapted to the measured data set. The computer 
program may be written in any suitable programming language, such as C++ and may 
be stored on a computer readable medium, such as a CD-ROM. However, the computer 
25 program according to the present invention may also be presented over a network such 
as the WorldWideWeb, from which it may be downloaded into the working memory of 
a processor. 

It may be seen as the gist of an exemplary embodiment of the present 
invention that a location of a portion of a structure on or of a moving object, which, for 
30 example, is not discernible in an image representing the original measured data set is 
predicted by using a model adaptation. For this, a model of the structure of interest is 
adapted to the structure of interest in the original data set. Once the model is adapted to 
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the image, portions (a location, a dimension or a direction thereof) of the structure not 
discernible or visible in the original measured image may be indicated by the model 
which may be displayed in an overlaid fashion on, for example, a display. 

These and other aspects of the present invention will become apparent 
5 from and elucidated with reference to the embodiments described hereinafter. 

Exemplary embodiments of the present invention will be described in the 
following, with reference to the following drawings: 



10 Fig. 1 shows a schematic representation of an image processing device 

according to an exemplary embodiment of the present invention, adapted 
to execute a method according to an exemplary embodiment of the 
present invention. 

Fig. 2 shows a flow-chart of an exemplary embodiment of a method 

15 according to the present invention. 

Fig. 3 shows a schematic representation of a coronary model according 
to an exemplary embodiment of the present invention. 
Fig. 4 shows a projection of a geometric coronary mean model with 
branches corresponding to Fig. 3 according to an exemplary embodiment 

20 of the present invention. 

Fig. 5 shows a sub-set of branches overlaid on an image and a 
corresponding area at a well-fitted model, according to an exemplary 
embodiment of the present invention. 

Fig. 6 shows a flow-chart of another exemplary embodiment of a method 
25 according to the present invention. 

Fig. 7 shows images of a coronary vessel tree generated in accordance 
with the present invention. 

Fig. 8 shows a flow-chart of another exemplary embodiment of a method 
according to the present invention. 

30 



Fig. 1 shows a simplified schematic representation of an exemplary 
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embodiment of an image processing device in accordance with the present invention. In 
Fig. 1, there is shown a central processing unit (CPU) or image processor 1 for applying 
a model of the structure of interest to the original measured data set, for performing an 
adaptation of the model to the data set and for performing an estimation of a location of 
5 at least one portion of the structure by using the adapted model. The image processor 1 
is connected to a memory 1 for storing the data set, for example, a plurality of images 
relating to x-ray angiography images. The image processor 1 may be connected by a 
bus-system 3 to a plurality of periphery devices or input/output devices which are not 
depicted in Fig. 1. For example, the image processor 1 may be connected to an x-ray 

10 scanner. However, the image processor 1 may also be connected to an MR device, to a 
CT device, to an ultrasonic scanner, to a plotter or a printer or the like via the bus- 
system 3. Furthermore, the image processor 1 is connected to a display such as a 
computer screen 4, for outputting information and/or images to a user. Furthermore, a 
keyboard 5 is provided, connected to the image processor 1, by which a user or operator 

1 5 may interact with the image processor 1 or may input data necessary or desired for the 
segmentation process. 

Fig. 2 shows a flow-chart of an exemplary embodiment of method 
according to the present invention, which may be used for operating the image 
processing device depicted in Fig. 1 . 

20 In the following, the present invention will be described with respect to 

the determination of the anatomy of the human coronary artery tree. In this connection, 
the present invention is particularly suited to improving and speeding up the diagnostics 
of coronary artery defects and their treatment Thus, for example, the present invention 
may be applied as an intervention support system for 2D-3D x-ray angiography. Also, 

25 the present invention may be applied in conjunction with catheter intervention. 

As may be taken from Fig. 2, in step SI, an image is acquired, for 
example, by x-ray angiography. This image may relate to a particular phase of the heart 
and to a particular projection angle. Then, in step S2, the model, which may for 
example be a statistic model projected into the plane of the image, is used to measure a 

30 distance between parts of the model projected into the plane of the image and 

corresponding structures in the image. This distance measured in step S2 is used to 
determine a similarity value in step S3, reflecting a similarity of the coronary artery tree 
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depicted in the image and the model to be adapted to the actually measured coronary 
artery tree in the image. In case it is determined that the similarity in step S3 is lower 
than a pre-set threshold value, the method continues to step S4. In case it is determined 
in step S3 that the similarity is above the pre-set threshold value, the processing ends 
5 and the image may be displayed to a user in a fashion where the adapted model is 
overlaid onto the image, allowing that the user recognizes where, for example, a side 
branch of an artery is to be expected or where, for example, a branch of an artery should 
be. In case the model, for example, indicated that there should be an artery portion, but 
in the actually measured image there is no artery portion, the user may assume that, for 
10 example, there may be an occlusion in the artery, such that no contrast fluid in the 
blood makes the artery visible in this region. 

As may be taken from steps S4 and S5, parameters of the model are 
varied. Then, as may in particular be taken from S5, positions and connections of the 
model may be varied and parameters such as heart phase 9, the projection angle/? of the 
1 5 model and an individual derivation from the mean model given by cp and/7 may be 
varied, which is indicated in step S6, allows a prediction of model positions by 
calculating a projection of model positions. In other words, in step S6, the three- 
dimensional model varied and adapted in the preceding steps, is projected into the 
image plane of the image acquired in step SI. Then, as indicated in step S7, the 
20 projected image is used to re-enter the iteration in step S2. 

The coronary model used according to an exemplary embodiment of the 
present invention may, for example, be taken from Figs. 3 and 4. Fig. 3 shows a 
schematic coronary model (topology only) with LAD, LCX and RCA as well as some 
sub-segments according to an exemplary embodiment of the present invention. Fig. 4 
25 shows a projection of the geometric coronary mean model with branches corresponding 
to Fig. 1 according to an exemplary embodiment of the present invention, as it may be 
achieved in step S7. 

The geometry of each branch of the geometry model is represented 
parametrically in such way that it expresses the position of a branch both in global and 
30 local coordinates. In the local coordinates, the position of a branch is defined relative to 
another branch, for example, to a parent branch. The parameters of the model are 
modeled statistically such that a given parameter setting is assigned a probability value. 
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A specific potential geometry of the coronary tree, as indicated in steps S2 to S7 in Fig. 
2 may be predicted by a parameter setting. Movement and deformation of the branches 
caused by heartbeat and by respiration are covered by the model. For the heartbeat, this 
means that geometry predictions are possible for any heart phase. To illustrate the use 

5 of the coronary model in more practical terms, an exemplary embodiment of a 
parameterization and a sampling is described in the following. 

According to this exemplary embodiment of the present invention, each 
branch is assigned a set of sample positions s b , . Each branch b is interpolated through 
its sampled position s bn by a spline. The model holds a statistical description of reached 

10 sample position, for example, a mean value s bi and covariance matrix. These values 
may be referenced to a global coordinate system, or they may be positioned relative to 
the corresponding parent branch (i.e. global or local coordinates). Also, information 
about lumen diameter of each branch may be provided by the model by l bi . The 
deformation of the coronary tree is covered by a variation of sample positions over the 

15 heart cycle <p = 0..271. Similar to the spatial sampling by sample positions, some 

temporal samples Sb,u with f=i..m are part of the model, where a temporal interpolation 
of the distribution of positions is provided between the m temporal samples. Thus, the 
model according to this exemplary embodiment of the present invention may be used to 
predict the position of each branch b at a given heart phase cp by a statistical description 

20 of the sample positions s bU and by spatial spline interpolation. This will be described 

in further detail in the following. 

For a given heart phase q>^ and a given projection plane orientation p & the 
projection of the mean model P$(^ g ,P g ) may be calculated by 

25 where M D is the matrix that projects a point onto the plane with 

orientation p g . 

Fig. 4 shows such a projection using an orientation similar to the 
schematic view in Fig. 3. This simulated projection may be overlaid to an angiographic 
image as indicated above. 
30 However, individual properties of the patient's heart may deviate from 
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the mean model s bJ ^ g . Also the given parameters like the projection orientation p g or 

especially the heart phase <p g may not precisely reflect the image acquisition situation. 
Thus, the model should be allowed to warp itself in order to find a good congruence 
between model branches and visible branch position in the angiographic image. 
5 According to an exemplary embodiment of the present invention, this 

may be done by applying a deformable model approach where the model parameters 
S bJ j m , q> m , and p m are varied to optimize a similarity measure cr = d m + f m (l) (see steps 
S2 S3 in Fig. 2). 

This similarity measure may be given by an image feature term f m (l) 
10 which determines how good the projected model branch positions agree with features in 
the image that correspond to coronaries (see step S2 in Fig. 2). A good proposal for 
f m (l) is a measure of the directed gradient at the predicted vessel boundaries. It gives a 
maximum value, when predicted coronary positions in the projection coincide with real 
vessel walls in the image of the original data set. At both sides of the spline through all 
1 5 sample proj ections of a given branch P s ($ g , p g ) = { s bg M • M Rg \ and with a distance l bgi 

to it, a strong image gradient orthogonal to this line is expected. It is measured at a 
couple of sample points and all values may be averaged. 

According to another exemplary embodiment of the present invention, 
f m (l) corresponds to the sum of gray values within the area of all predicted projected 
20 coronaries, assuming that coronaries appear bright in the image. For each position in 
P s (<fi g ,p g )= { s b f f g • M Pg j given by the actual parameter setting of the model, a circular 

area Q/ with diameter is taken. The unification of all areas U may be given by 
U = ^C bt . The average image intensity on this selected part of the image yields the 

feature term 

J7(x) dx 

25 f =^ 

'grey 

U 

Fig. 5 shows a sub-set of P s overlaid on the image and the corresponding 
area U (union of circular areas) at a well-fitted model in accordance with an exemplary 
embodiment of the present invention. As may be taken from Fig. 5, Fig. 5 shows an 
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example of the area U where the image intensity is measured. By using, for example, 
steerable filters on the image at the modeled position, both image gray value properties 
and image gradient properties may be combined. However, as a downside, using such 
steerable filters increases computational efforts required. 
5 The distance measure d m of the parameter setting to the modeled 

distribution is considered in the similarity measure. There must be distance measures 
for each parameter or parameter set: 

for each b and each i, 
and 

d p (p m )=\\p m -P g \\ 

where ||x|| is a suitable distance measure in the domain of the parameter. 

As the model holds a distribution cf, for each parameter s bJ ^ also a statistical 

15 distance measure (e.g. Mahalanobis distance that also considers the covariance matrix) 
is applicable. A weighted sum of the three single distance measures is used as distance 
measure of the model parameters to the expected parameter setting 

bj 

This term introduces constraints by the a priori knowledge about the 
20 expected shape of the vessel tree projection and will penalize parameter configurations 
that might fit well with the data but that are very unlikely to reflect an existing imaging 
situation. 

According to another exemplary embodiment of the method according to 
the present invention, a segmentation step to find the main branches in the angiographic 
25 image precedes the adaptation. It yields a measure of the center lines of successfully 
segmented coronaries called S. The subsequent adaptation is computationally less 
extensive, because no feature term has to be considered during the adaptation and no 
access to the image is required anymore. Instead of the image feature term f m (/) a 
distance measure S m (S) on the segmentation result is used: <j = d m + S m (S) . S m (S) 
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determines the distance between s 6 ^ • M Pm and the center line of the segmented 

coronaries in the image. This variant also allows to use different model candidates and 
to decide, which one fits best. There could be for instance a model for left-dominant 
type and one for the right-dominant type. 
5 Instead of a global measure for the whole coronary tree as those 

introduced above, also a part of the model representing a single branches or sub-trees 
may be adapted to an image. The position of such a sub-tree may be modeled 
statistically with respect to its parent branch. Thus, the given or previously estimated 
position of this parent branch may be used to predict the position of its desired sub-tree 
10 in the image. 

Instead of using a set of template models, each with a given geometry, a 
deformable model is used that allows its geometrical parameters to be varied within a 
certain range of values. Once it is adapted to fit the available data, a geometric 
prediction 3D and even in 4D (space + time) is available for the whole coronary tree. 

15 The range of parameters is determined by the modeled statistical distribution of the 
parameters, but also by the result of the feature term f m (l) applied to the image - or 
S m (S) applied to the extracted center lines respectively. 

Thus, the model may be adapted to an individual image wherever 
evidence is sufficiently available from the image. In image regions where evidence 

20 given by f m (/) is low, a priori information d m from the model is dominating the 

adaptation result This advantageous property of the model and the application of the 
model in accordance with the present invention allows a well-founded estimation of 
branch position taking into account available knowledge about proximate branches. In 
the same way, a trajectory of a branch in time may be predicted from one or more 

25 images taken at different heart phases. The geometry at an unknown heart-phase may 
be estimated from the model taking into account available data from those heart phases 
for which images are available. Thus, advantageously, for example, patients with 
chronic total occlusions (CTO) in coronary arteries, the whole vessel branch beyond the 
CTO is usually not visible in angiographic images, because no contrast agent is 

30 transported there, can be diagnosed more accurately and easily. 

Fig. 6 shows a flow-chart of another exemplary embodiment of a method 
according to the present invention. This exemplary embodiment of the present 
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invention is in particular suited, for example, for diagnosis of CTOs in coronary arteries 
where the whold vessel branch beyond the CTO is usually not visible in angiographic 
images because no contrast agent is transported there. 

The two-dimensional images of angiographic images makes it hard to the 
5 cardiologist to predict the position and extent of coronaries. It is especially hard to 
differentiate an occluded ending (CTO) from an ending that extends further but 
orthogonal to the image plane. Reasoning and comparison of several projections from 
different angles is usually required with this kind of differential diagnosis. Therefore, a 
CTO may easily be missed unless it is explicitly suspected and all potential image 

10 positions are scanned by the observer. This situation may be improved by this 

exemplary embodiment of the present invention depicted in Fig. 6, where a simulated 
projection of the coronary tree is shown to the user together with the angiography. Due 
to this, a CTO case becomes obvious by a branch indicated as missing. This will be 
described in further detail with reference to Figs. 6 and 7. 

15 Steps SI to S7 of Fig. 6 correspond to steps SI to S7 in Fig. 2 and thus, 

to avoid unnecessary repetition, reference is made to Fig. 2 for the description of steps 
SI to S7. 

In contrast to the method of Fig. 2, from step S7, the formally three- 
dimensional model, which has been adapted to the image acquired in step SI, i.e. which 

20 has been adapted to feature in the images acquired in step SI and which has been 

projected into the image plane of the image acquired at step SI, is overlaid in step S10 
onto the image acquired in step SI (an interruption of the adaptation circle of steps S2 
to S7 to branch off to step S10 may be decided in step S3 when a sufficient similarity 
has been reached). Then, the image is displaced in step SI 1 to an observer. Due to this, 

25 the image visualized in step S10 shows the coronary tree or the respective arteries 

discernible, i.e. visible in the image and the overlaid adapted and projected model. Due 
to this, for example, a CTO case becomes obvious by a branch indicated by the model, 
but missing in the image. 

This is depicted in Fig. 7. Fig. 7 shows three different images generated 

30 in accordance with an exemplary embodiment of the present invention. The first image 
indicated in Fig. 7 by a shows a main artery (black), which is clearly visible in the 
image. However, as may be taken from image a, a proximal part of a coronary vessel 
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tree is visible in this angiographic image (black) and a distal part, which is not visible in 
the angiographic image and which lies beyond a total occlusion is predicted by the 
model (white). As may be taken from image b in Fig. 7, the main branch has an 
occlusion and would usually not be visible in the angiographic image. However, the 
5 occluded part of the main branch is indicated by the model. Image c shows a functional 
occlusion. 

In other words, for CTO diagnostic imaging, the model is adapted to an 
angiographic image in the way explained above. A simulated projection of the model is 
then overlaid with the angiographic image. A fused visualization of a measure (the 
1 0 image) and the expected normal finding (the warped model) immediately allows the 
cardiologist to diagnose a CTO (see Fig. 7) at positions where a predicted branch is 
missing in the image. Especially in screening situations and in cases without previous 
indication of CTO, it appears hard to visually scan all relevant positions. By using the 
present invention as indicated above, finding CTOs may be significantly facilitated 
15 because differences catch the eye. 

The above explained exemplary embodiment of the present invention 
may, for example, also be applied in conjunction with catheter intervention, where a 
catheter is to be navigated through the vessel to an occlusion. In particular, in cases 
such as depicted in image a (Fig. 7), it is usually hard to find the entry point of a side 
20 branch having an occlusion, since the side branch itself is not discernible in the 

measured image. The visualization of the predicted position, as shown in image a (Fig. 
7), allows an improved navigation of the catheter. 

Fig. 8 shows a flow-chart of another exemplary embodiment of a method 
according to the present invention. As steps SI to S7 of the method depicted in Fig. 8 
25 correspond to steps SI to S7 of the method depicted in Fig. 2, reference is made to Fig. 
2 for a detailed description with respect to steps SI to S7. 

After a sufficient similarity has been reached, (which may be determined 
in step S3) the method may continue from step S5 to step S21. In step S21, branching 
off from step s5, a prediction is made by calculating projections of model positions for 
other heart phases <p and other projection angles p. Then, in the subsequent step S22, 
the warping parameters are determined and applied for overlaying images, which may 
allow for an image enhancement. The operation performed in steps S21 and S22 will be 
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described in further detail in the following: 

During an angiographic intervention usually a couple of images is taken 
from slightly different angles. These images can be superimposed to each other in order 
to enhance the signal of a given object of interest with respect to the noise level if 
5 corresponding positions and orientations of this object are known in the different 

images. Here, the correspondence is established via the coronary model. Corresponding 
positions and orientations are available and local image registration could be achieved. 
The projection of a sample point of a given branch b g is given by s b ^ Um • M Rm where all 

model parameters are adapted to the image. These corresponding image positions are 

10 used to achieve a landmark-based registration between all the images. In order not to 
superimpose images from largely different viewing angles, only projection with low 
differences in p g - or p m respectively - should be chosen. The superposition of a line in 
the images that is expected from the adapted model to be the location of a branch makes 
the branch visible even though it was not clearly visible in the single images alone. This 

1 5 should work for smaller sub-branches that are hardly detectable and for the case of 
functional occlusion (Fig. 7) where only a certain amount of contrast agent is 
transported to. But this technique is even promising for totally occluded branches as it 
also enhances the poorly visible vessel wall. The true benefit from the model-based 
approach comes by aligning the model to the visible parts of the coronary tree and 

20 predicting the position of not sufficiently visible parts from that. 

According to exemplary embodiments of the present invention, a model 
of the coronary artery tree is adapted to the coronary artery tree visible in a measured 
image. The estimated position and orientation of cardiac structures reflected by the 
model may, according to an exemplary embodiment of the present invention, be used to 

25 calculate a set of parameters, for example, for the x-ray equipment. Such set of 

parameters may include position and orientation of x-ray generators and detectors in 
order to produce the cardiac standard projection image or in order to achieve an optimal 
x-ray set-up to image a given coronary artery. Furthermore, an iso-center for rotational 
cardiac x-ray imaging may be determined according to this exemplary embodiment of 

30 the present invention. By continuously feeding generated images during the 

examination to the model registration method, the accuracy of the adapted geometric 
model and the estimated position and orientation of the cardiac structures improve 
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further, especially if additional structures (for example, additional coronary arteries) are 
imaged. 

Advantageously, this may allow for a very accurate setting of the x-ray 
equipment and a reduction in the x-ray dose applied to the object of interest, i.e. the 
5 patient. This may also allow to reduce the amount of contrast medium applied to the 
patient. 

In spite of the fact that the present invention was described above with 
reference to the human heart and the determination of a coronary artery tree of the 
human heart, it is obvious to the skilled person that the above described present 
10 invention may also be applied to moving objects in general. In particular, the present 
invention may be applied to the determination of structures of a moving object from 
two-dimensional data sets, where, for example, the data set comprises images taken 
from different projections or taken at different points in time. 



